Chapter XIX: Multi-Modal Biometrics Fusion for Human Recognition in Video

This chapter introduces a new video based recognition system to recognize non-cooperating individuals at a distance in video, who expose side views to the camera. Information from two biometric sources, side face and gait, is utilized and integrated for recognition. For side face, an Enhanced Side Face Image (ESFI), a higher resolution image compared with the image directly obtained from a single video frame, is constructed, which integrates face information from multiple video frames. For gait, the Gait Energy Image (GEI), a spatio-temporal compact representation of gait in video, is used to characterize human walking properties. The features of face and gait are extracted from ESFI and GEI, respectively. They are integrated at both of the match score level and the feature level by using different fusion strategies. The system is tested on a database of video sequences, corresponding to 45 people, which are collected over several months. The performance of different fusion methods are compared and analyzed. The experimental results show that (a) the idea of constructing ESFI from multiple frames is promising for human recognition in video and better face features are extracted from ESFI compared to those from the original side face images; (b) the synchronization of face and gait is not necessary for face template ESFI and gait template GEI. (c) integrated information from side face and gait is effective for human recognition in video. The feature level fusion methods achieve better performance than the match score level methods fusion overall.